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; . INTRODUCTION \ : P 
\ ; r} = 2 : 
’ Compared with the U.S" total expenditure for educatina : 


\ 


: ss 


our youns, which has-been called our larsest national indys-~ 
, Tyr the amount spent on educational research... is pitiably’ 


tai? small. All the more reason, then, that the present Federal 


Administration, committed as it is to reducina the budset 


wherever possible, should look closely at each matiof commit— 


i « 


; : ; ‘ 
ment in educational research, to appraise the past oPera- 


. : : : . ~' ; 
tions, with an‘eve toward improvins the future, and of 


‘ ‘ % * : N 
course to eliminating.any prosrams which are not, Shee sere 


tive. . 7 ’ . 


The National Assessment of. Educational Rrosress (NAEP) | 


would, naturally, come under important review. It had its 
‘ a : ' o ‘ 
orisin nearly 20 years ago, in’a political and, ideolosical 


. ° 


(and sodiat science) atmosphere auite different from to- 


day’s. It has cost about GO million Federal dollars, plus 


another 5 million from private Funds (and huse uncounted ex- 


Penditures in professional and student time at the state and 
= ’ , 


. 


¥ 


‘its present. Form? 


. From SEA and LEA personn 


local levels). There’ are, then, very valid auestions which ~’ 
we may Sey 


the opératins assumptions of ;20 years aso 


~~ 


still the Cia oetind Should it, indeed » survive in 


nd if sor what changes should be ‘made in 


te 
ry 


.its desian stratesy, its samplins, “its measurements®.its re- 


Portina, its administrative procedures, to improve its use~ 
fulness? ‘ ; 
‘Other Studies of. NAEP haver, of course, been conducted in 


« 


the Past. Some maJon studies have been rerovted by freen- 
baum (1976), by’ the Nal ional Center For Education Statistics — 
(NCES, 1974, 1975), y the General Accountina Office (GAO, 


1976), by Wirtz ard Lapointe (1982), and by Sebring and Bo- 


-ruch (1982). These A been useful in drawing’ attention to 


i ? 
certain’ accomplishments and apparent inadequacies of the 
‘ : ; . 
NAEP, operation and dissemination. Most of these have taken 


« s fy 
. 


NAEP ‘s Founding principlés as a siven, and some have lauded, 
NAEP‘s ‘methodological contributions to the saitastiane ce 


- 


knowledase. A persistent Plaint has been about NAEP’s pPossi- 


ble inadesusoies asa useful and active aid to the state 


wid local diuiriavs (SEAs and LEAs). These nesative com 
ments about 4 oi use have been aa a by Sebrins 
and Boruch. (1982), dho have pursued inForMaeion or testimony 
: ‘ about NAEP usefulness, and From | 
NAEP personnel about their enceuatels at other educational 


° 
. 


4 %, ry 
levels. - - : os : 


In commissionins the present. report fone of Pea je brief 


’ evaluations 50 commissioned)» ‘the National Instituterof Edu- 


e 


cation (NIE) explicitly asked: fg 


e 


° 


‘ 
? é A 8 : « 


How can, the NAEP desisn: be —enhanced to allow For the 


4 « 
generation of information useful tor. ~ 7 
. $ J ‘ 


Ig: 


A. SEA and LEA and other state Bolicymakers 


B. Federal policy makers” 


C. Professional associations w ae 
A . a ‘i x « > ze : 
» . OD. Research community : et 
' : % ; . a : e 
E: The public. r : et ; 
&. os 7 : i 


This report will make a share break with the past evalua> 


. 
. 


tions of NAEP, I intend beina much more critical,’ of the s 
. ? ns LY e e ‘a a re ae 


Founding principles underlying NAEP. I will arsue that 


. é * x 


these principles were often rooted. ath ihe ‘Political and | 


4 


> AS . ee 
ideolosical commitments oF the mid- 19G0's.- + Such “a eondt-" 


se 


tion, if true, surely Acetate in itself invalidate the 


‘ 
. 


principles. “But I shall further arsue that these princi 


+9 ; a 


‘é 


Ples, however fashionable they may have been ‘at the timer 
aed j ges 4 : : . ware ® . \ 
were at times cripplins throush the employment of poor sam- 

. ee e re ® , 6 
pling, measurement, and resedrch desian. Many of the more 


intellectual problems-of NAEP (as ‘distinct From the practi- 


2 9 8 


~ 


cal problems) apparently stem from such poor principles, 


| themselves rooted in Anadeauate scientific ‘yiderstandinas. 


a . 


” 


a 


id 


e . 
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. nn , 
..'_, NAEP*S IDEOLOGICAL, UNDERPINNINGS 4 


e oo 


As most educators are auarer 1957 marked a strona shift 


ey ‘ o _@ : F 
‘toward Federal support for education. THE, SEMEN at that 


‘were not liberal but conservative:/ Russia 


~ 


vehicle, 


timer 
= 


t 


into orbit its Sputnik’ space. ‘and America, well 


\ 


eS : ® ta ; Ban Sets , : 
.aware of Russia’s expansionist military sive and srowins 


-nuclear threat, saw education in ay neu light. «Suddenly, 


there were srave national reasofis 
- ~ 


For assurina U.S. 


or superiority in scientific and technical education rele- 
. : * ‘i ic wee 4 : ¥ . : 
vant to military security, . Indeed, the “1960 RPS SSR EEAX 


campaign Was marked by debate over the best ways to aartete 
; ‘ 
“Kennedy, 


a f 
such seourity, John. E. 


and the vietor, 
‘ > 5 


that the Former administration ney, rermitted a’ "missile 


the mood of Consress in the late 1950’ s “was 
e - ‘ 
wien senanly ‘Pro- - defense and. committed to strona Federak 


“involvement in education pirected to this Military aim. 


- sap." In Fact, 


One 


* reskit oF auch Conareéssional aemane was the National De- 


a "es 


Fense, Education Act. anothers, fap taitaey? eeteumnt to our 


° et 
-brieP history oF OEE. was he: ffoundins -oF Provect TALENT. ° 


. . " . ~ R 


the very nani differences betueen TALENT ang: NAEP are “Seo 
jue tien the sees apes 


thie Lifes. In ‘one case, 


Con 3 24 = ET ‘ . 
the nation sousht the identification ane nurturance of hish- 

1 ¢+ ° att es % 
ability students (ives, “talent") 


resource For. the natidnal strensth. th the other, the na- 


tion: considered not so much the talented individual, 


had put 


Parity 


had cieengd- 


as an important national - 


"as it 


a 


. 


ot 


¢ 
*sousht political and leaal- ways of improvina society as a 
’ ‘¢ 
Whole. In Project TALENT, the most important soal was the 


recosnition and cultivation of hish individual intelligence.” 
t 


.} ° . ; ' e e , # 9 


_In the case of. NAEP, intellisence hardly seems to have been 
an admitted reality (there are not even any.explicit intel- 


? qraene® “hgasures in. NAE?’s testing program). _ in TALENT, no 
ethnic igfermation was asked? asain, che individual was cen- 


tral. For NAEP, there was extensive inPormation goiiected 


° » ¢@ 


about race and SES, and other sroup variables:. and the em- 
phasis Seemed to be more on issues of “eddality," rather 
5 ‘ ae 


- 


. than of “excellence (cf. — JohneGardner’s influehtial book — 


N 
ie ory, : . . ta 
e . , a . ° 


6F 1961). é, + : ; 

; ? “ ‘iy : ; te . a 

* What had happened’ to shift the. emphasis so markedly? 
‘ ; one . , : 3 = aa 

* Followins theAlandslide election of L.B. Joknson in 1984, 


° + 
Ni 
nY p . ’ - oe . 
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‘ t 
‘ «2 ; 
, especially by laws and peauldivons favoring ST ouPs resarded. 


. vs 
~ z ' 


“as, having suffered dientdeinerine in the rast: Blacks» i! - 
> : 


Lae 


td . 
tive Americans"), and women of afl’ ethnic Ssroups. Universi- 
m ty’ *sampuses> of course, are often the- “breedégs sround for 
. 4 y . 
-ideolosies& and 6uch ‘was clearly i bis pase during the 
‘ tye s ; ry , 7 . 4 = 


mid-1960's. The spirit of revolt Found student (and Facul- 


~ 


CL eet Py 2 ’ * 


. ty) expression asainst the military draft, ‘asainst-Perceived 


r ee ‘inequalities: and even -- perhaps especially -- atainst the 
i ? “4 y ‘ v e * 


tosether with a liberal Consress, America moved. toward an 


‘unprecedented Federalization of social proarams, ' marked’ 


tinrAmerieans in the U. S., ‘Indians (who were venamed *Na~ 


iv r ‘ e i . ¥ . 
‘campus “Establishment” itself. There was a. widespread» and - 


aa 


. F . ta 


very well-publicized, breakina down oF Former social rest- 
. , ' 
faints about public ‘and private behavior "including social’, 


gexual,y sartorial? and Political behavior. In seneral, *re- 


® U . * 
Form was not expected From almost any traditional centers of 


. tracapes ct, : F ) 
tered. One heard the expression, % trust anvone. over 


. “ 


Lo thirtyi | | es 
\ ‘ ; : ‘ 
“What was meant, then, by President Johnsons call for a 
“Great Society"? It was to be a society in which inequalit 


/ 


was to be prosressively reduced thrdush the gperation of the 


larger society itself -- and especially. throush that’ super- 


. 


ordinate level, the Federal ‘Government. There was an ehor- .- 


of ; 3 
a" \.mously increased level of Federal taxation, of Federal ’bu- 


. 


reaucracy, of federal lawmakina affectins the conduct of 


‘almost everyone, and most certainly of *the schodls. and col- | 


| 


. é \ 
leses throushout the land. gra 4 -\ 
: ; +o : : _ sae \ 
‘Politics and Social Science .., — ; 
a ws ; 7 \ e 3 
© fe ¥ ; \ 


goo What we ‘have been recallina, howevers seems purely a 
. : w e \ 
+ Guestion of Political ideolosy,’ ‘with hisete bearisns on the 


na 


structure of. what should*tte, in principle, a scientific en- 


ment and “pesearch have close Kinship .with btatistics and 


e , 


mathematics. * Fields of an abstract, .universal truths And 


when these methods of science are aprlied to the observation 
. fe. . 7 a : * ¢ i 
and educational phenomena: then the empirical ob- 


of social 
«b ‘a 
; «7 of 


terprise.. After all. educational and psycholosical measure 


authority, unless these were gaken over dnd. radically ade 
<i f ’ . ; 
scdasanrth ? “ . ; 


. 


ae 


~ 
a 


. 


from such wtethods We should be ina eeedeian to discover 
: ee) 
educational "truth," as well. l a: 
we is s Ve ° ’ ‘ 


While such an ideal operates in principle, in the real 


world of the mreiied social scienges, “the methods and be- 


. b + ri 


liefs are often servant! to the overarchins idea lane inelud- 


ins most definitely the"rolitical (dendonys and any other 
’ - . ‘ . b 
beliefs Favorable.to one’s own well-beinar or that of one’s 


on . 


‘colleasues. “In psychology, a example, a larse number of « 


one’s ‘colleagues earn a ved Boe through she practice of 
"psychotherapy" -—— douwite the Pees of any unified theory. 
about. its operation or of any ererren suddenne about its 


. 


efficacy. With the éxcerptiaon of a Few mavericks: often dis- 


: . we a 
Pparased by their Fellows, ‘few eminent psychologists have 


, a 


seriously addressed this serious problem.’ # |. 
. P’ e 


ry 


. 
- . * bs 
, ‘ & e 
‘ . ‘ . 
* * 


And the trainings of social scientists seldom includes 


‘systematic instruction in sciéntific ethics. — Few textbooks 


f . 


of Teseargh methods make any ‘mention °oF what one could. “are 


would be the premier assumption of the scientific enter-. 

Prise: That we attempt to - discover, only the truth, ae that 
we expose and decry Falsehood. "Rather, in undvenss* ¥, train- 
ing for advanced. desrees in. psyohalosy, sodioloay, and simi- 
iar Fields, a hisher virtue is. Sometimes fTesarded as the 


. | e ’ 
adoption of a liberal intellectual posture. Such a casual 


ean 


observation ‘is indeed sue orted by careful research with 


servations, and even their” interpretations, should Follow ° 


» 


political S4uestionnaires: ‘Ladd and Lieset (1975) have re=- 


- 


Peatedly confirmed that the "social sciences," broadly de«. 


« 
- ‘ a 


Fined, ‘are the left-most maJor sesment of academigo faculty. 


Thus we reach the vexed problem of the, relation’ bet een pol- 
. * * Y A ; 2 . 
itical’ ideolosyr on the one hand, and the conduct of scien-" 


tific research, on the dther. 


: , 
Environmentalism and Behaviorism , 


> ; 
«There is a complex of issues: in which political. ideoloay 


J interacts with psycholosical theory. One of these key areas 5 


» involves the relattve emphasis on environment (us. heredity) ~ 


in its inf luepwe on behavior. In the major Field of “learn— 


ina theory," as Sauahs during ‘ the’ 1950’s and into the 
* es * 


1960 $3 -human diPremenass in learninsa were ne eee treated (in 


ws | . 


the eoravduenval literature) as an Saran eerne in the ana- 


| - 
Lysis of. data. Indeed, to Justify the. huse theoretical 


leaps between ‘the laboratory . animals and eat in 


ae 


. * ra } Pa Be ae 
schools, it was common tq-learp even the differences between 


ae 
= 


species. At the-same time, of churser with the huse exran-- 
’ ‘ * . a 


Field of Perphowetrics Was looking exactly at. ‘hea: husan. 


differences, and makina ‘sreat strides. +hat these ' two 


' 


Fields, psychometrics and experimental eevehilesy: had lit- 


tle to say to each other was pointed out by the, seuoRe ona 


rsycholoaist Lee Js Cronbach, himself a SS heater of 


- v . , *. 
note, in his Prasidential Aitvess oF the Averigan Psycholo- 
° ou / 7 . © ie a: a ae ; . ; 
‘ e > 

‘ ee net Be * thee 

, a : 
™~ 
‘ : nen ON, °F {0 
ry “* 


. 
. , i 
. . A 
. & \ 
: ; os 
. 


sion in testing and the s@dden access tq ‘computers,’ the - 


< 


ge ‘gical Association, entitled "The Fic Disciplines of Scien- 


* tific Peycholoay." (His technical solution was that the 


analysis of covariance could combine human differences and 


é 


experimental effects into a sinsle statistical: treatment:~ 


. 
, 


50 much For the underlvyins theoretical clash!) 


a 
*. -, ie o ‘@ . = 3 7 


a 


+ It if not‘surprising, “then, that the swing of ideology 


or ° . : . : 
would cause a corresponding swing.in Fashions amonsa applied 
a lw 


o ¥ ° ; 


Y social scientists, From one of these ‘emphases to the’ other. 
‘ . ; « © ca \ 
_ ' and aie activist 1960’s,' when many were lookins toward 
: és 


- 


sovernment to resolve many human problems) a very persuasive 
fo, a é ¢ 


>, . SPaNSENAD Tr environmentalism was already present, in the 
* > 2 noted exrerihentalist B. F. Skinner. , , 


es % : > s é ~ . ba \ 7 4 Be +. : 

Sr /RBdical! eTyironmental Fsm, -- the emphasis on trainins tb 

the exclysion'ef individual differences -- had of course an 
2 * . ¢ 


’ ‘ 


implied that the nervous system was a clean slate at birth 


. 
° 


(and of course one “clean slate" is like another). John B. 


~ 


Watsan’s vaunted claims for ‘behavioral ,"conditipnina" Over- 


' ry 4 


rode any innate differences in talent or disposition, sand 


e 
Lg 


assumed curreht ‘mastery (even in the 1920's) ‘of training 


techniaue. , B. F. Skinner'was the articulate inheritor of 4 \ ” 


“J this behavioristic tradition and -aureleLea wist dneiidnsial 


~ a 


galing araduate students, he “pelesated even learnings theory 


’ « 


« to the ashhear of the unnecessary (Skinner, 1968). And-with 


a 


ancient~and resectable tradition. ‘John Locke’s tabula rasa 


~ . advocate in the history of psychalosy. To the Joy of strus- 


- 


ee ee ee eS oy > 3 aM - 
A . : . 
- ~ . . 
. ; . 
. . 
7 . . ay” - 
3 e 
E . ig e . 
. 
. 
a ’ 
yo ak » A 0 
‘ : 4 he = [sage 
‘ ¢ rar 


@ 
. - 
. . 


his sweepins vision of Virtually all behavior as under the 


control of what he termed “operant: reinforcement" he sener-.« 


‘ * 3 
fe ae alized From rats in a SKiriner boxr to “piseons in a Pelican" 


(1971), to the productive and well-behaved Fictional charace 


‘ ° ? 
* 


ters in Walden IT (1948). ©Skinner’s followers were lesion, 


ate departments in psycholoay and educational psycholosy 


with easer and apparently successful practitioners. 


" . e 
f y r) 
k . 


. 


4% 


uf aggression, to learnina the alphabet, to mastery of some 


school skills. Borrowits heavily From the methods of “task 


é _ 


analysis" (e.s., Gasne, 1970), the behavior mod enthusiasts 


listed clearly the appropriate stimuli and the desired res- 


. & 
Ponses, elicited these responses from,their students fol- 
lowina the stimuli, and then rewarded the students (the re- 


* ‘ ; r ead P 
ward was the i la of the S-R pairings), 
i 


‘ 


i and. repeated the exercises with Progressive "“shapins" oF the 


' 
. 
’ : € 


havior mod represented, Fit ald into the’ Sovernment smood of 


- 


i "th® 1960’s. If by chansins the environment in this Ways by 


aaa 


e > > 


Providins ae (often one- md 5 one) conditionina to the 


- learners, ana by usina systems of enjoyable reward, we could 


“ef fectivaly educate eyen ‘low-achievins populations, then we 


and as specialists in “behauior mod," they Populated sradu- 


. 4 , 
And their apparent suc@esses ransed From the extinction 


ie ot eae ° te 
, ee sae the desired resronses, which were the “cré- 
‘ _-te ion" or “obJsective" of the prosram. oe 


° 


7 7 # —_ ome 


3 Such environmentalism, the radical behaviorism that be- 


“miaht eliminate isnorance and. Poverty ane iiebennoder: _ one - 


. 


o 


seneratipn, And the devélopment,.of computers ‘and special 


textbooks provided even an economical way.to do all this, 


os , 
4 ; . fo) > , ; ce . e* ar 
throush “prostammed instruction. ut what was required, 
z First and. foremostr ‘was to jeati out ll * cai @viguiues : ‘ 
is 7S pesponse. pairings whieh would constitu e the broadly desira- 
rs Z . 7 ari “ . 
- » ° ble behaviors which MEP iia objectives’ of the entire educa- 
; “tional enterprise. | Tf the Federal Government * could hele 
Q soe. bring ‘this about, then it surely should (an assumption Bins : 
on a loosely andar ane equality, ‘oF opportunity” mandate). And 
‘ ‘ ; 
bi. 8 a therefore .the 1960’s and 1970ts saw a‘powerful surge of Pro- 


grams aimed at such "behavioral technolosy," and funded by 


ee y 5 « as . 
federal asencies. ee — 
4 : . 


. se . * ‘ « 


‘ ‘ t ' 
Unfortunately Panes (anima of suc n enterprise, how- 


‘ 4 


. ever, its advocates were still not communicatins with the 


‘7 measurement scientists. And when they did, they Felt embarm 


y 4 
‘ rassed bx the irreconcilable nature “of isome of the underly- 


. ‘ - 


2 Y ins theofies and practices From the two Fields. Thus, for 


any persuasive. rationale For behavior mod, it was necessary 


Pay — 4 


to ‘develop a contradictory type of measurement theory. And 


4 


ao, 2 fe fs 
this need was: the Powerful ensine of the drive toward test- 


“ 


ins which would be .the “criterion” or "“obJective" fof the 


. 


prosram. ‘ 


‘ 


‘SELECTED ISSUES OF NAEP ‘S STRUCTURE 


° 


As we have seen, the ideolosy and political soals of the 


mid-1960's seemed to demand a set of social science concer- 
tions which would rationalize these s0als and permit their 


—s P 
realization. A number of central issues of NAEP’s orera- 


tion, then, were resolved in compatible Ways. NAEP would 
emphasize fhe importance of denvironment, and diminish the 
importance of heredity. NAEP would emphasize the item-by- 
item learnings of ‘skills, ranher than the (less easily al- 
tered) operation of seneral intellisence. NAEP would con- 
centrate on the educational prosress df rr and sive 


more impdrtrance to their equality than to ‘individual ta- 


lents. ‘ L ae 


To achieve worthwhile goals, it is not always necessary 
to employ sound theory. But in the recent decades of social 


/ : a a: 
science, Poor theory has often Ted to results that were 
{ 2 


al 
r 


wasteful or worse. And several central issues From NAEP’s 


« 


history show weak theory which apparently has hampered NAEP 


t 


in achievina maximum utility an@& explanatory Power. 


re 
‘ 


“dbdcotdue- US. Norm-Referenced Testing 
‘ ‘ . é 


‘ PY 
If behavior modification.were to appear successful, then 


it must appear to achieve GR tated objectives, the chanse 
in behavior desired. If we ate soina to make claims of 


success for a new prosram of teachings readina, For example, 


oo 


oe — | : ) : ye 


e ne ‘ 


then we should be able to Point to S“R pairinss ‘hich faba 
aoe f ; SE 
: been clearly altered in.the desired way. The trouble is, : 


. standard ability and achievement tests, unless they are 
or “ , . 


themselves practiced durina instruction, are. not apt to:be 


ry . 


easily altered by behavior mod eroasrams of reasonable lensth 


‘ 
and expense. This diFPicylty was a severe one for applied 


. 


° behaviorism -- and there were in seneral Just: two solutions 
i ig it.’ One was to continue the use of. standardized tests, 


but teach the items during the, behavior mod instruction. . 


The other was.to araue for new instruments,* specifically 
tarseted durins the instruction period, and to take these 


new instruments as the “obJective" of the educational pro-” : 


-aram. A third alternative was unacceptable: To use the 
vee ‘ef F 
standardized tests as criterda of the programs -- and to 


lose.any refundins for those prosrams! Here are two histor- 
‘ ‘ 4 
ical examples, each originated in the 19G0‘s: 


“ 
cl 


Teachinabhe standardized intelligence test: The Mi lwau- 


‘ 


kee Protect... -The lonsins for a strikinaly successful go- 
° ‘ » . : 


P , vernment intervention cannot be better illustrated than by ... 


‘ the "Milwaukee ProJece," directed by Dr. Rick Heber of the 


University of Wisgconsin-Madison. At the lona-run cost of ‘ 


over $16 miVlion/in federal Funds, Heber and his colleagues 


concentrated of just’ 20 disadvantaged children From a Mil-» 


waukee slum, and claimed I.Q. mean gains. of 35 points, com- ° . 


- . 


, * ‘ . 
pared with their untreated controls. As it happens, despite 


; ce ; . 
the theoetical and symbolic importance of this demonstra- 
- 2 « Cy ‘ 


. , * 


.. @Muestions of the claimed effects.) 


tion, technical reports were fFusitive and Plagued by ambiau-— 


w 


. 


ities. But the rare téchnical critiques of the ProJect 


. reached the inevitable conclusion: that the personnel had. 


been systematically trainina the younssters im the material 


‘cess (Pase, 1972b; Page & Grandon, i980). (As it happens» 


» 


Heber and one associaté were’ later convicted of misuse of 


- federal and state Funds and are in’ the: penitentiary, but the 


» ° 
~ 


published critiaues concentrated only on the scientific 
o~* Bi ta * ; : * 
_Even with the exrpendi- 


, 


ture of over half a million rer child, the Milwaukee ProJect - 


a. 


Pailed to produce measurable and seneralizable, Jona-ranse 


effects in performance of standardized tests of Fundamental 
‘6 . i] ; 
importance, when these were not trained by the ProJect it- 


“ 6 Pag . 
self. , ‘ » * 
2 


Teachina the standardized achievement test! “Performance 
eae ae Experiment. There rae better test of sehaveur 
‘mod, broadly conceived, than a larse experiment ‘oF unprece~ 
dented sweep, conducted, by the OFFice of Economic Opportuni- 


* 


ty (OE0) and reported in 1971 (Pase, 19724). "Parformance 
contractins" (PC) as a system of instruction through behau- 
ior eoaieiantioay whereby students and Psycholosist-instruc— 
tors were reuueded according to how well .the students ‘igs- 
tered ineenwardanal prosrams in peadina and matheamtics. 


é. ; c 
Ten thousand disadvantaged youths from thrfe :srade levels 


Ase Ne ona x 
and From many different national sites were trained by en- 


a 


of the standardized ¥.Q.. tests employed to demonstrate, suc— 


? 


- 


psec that the experiment is unknown to most studentg and 
r : 


. 


terprisins R »& D ‘corporations, accordins to the leading 


° 


theories of the behavior-mod school. All were optimistic 


about the outcome, - on the basis of apparent successes of PC 
4 


already “demonstrated in Texarkana. Unfortunately, however, ° 


that earlier demonstration was marred "by the discovery that 


students had been trained in the very items of the standard- 


ized tests used to evaluate/’the proaram. Hence, in this new 


larse-scale OE0 _ experiment. there eee no clouds oF 
doubt: Trainers were Ponetdden 46 eeLUge material -From the 
tests union would be used, under penalty oF losina their 
Pay. Bid the standardized tests were administered by riso- 


rous evaluators unconnected with the R & D corporations and 


‘with the schools. The results wére shattering -- so’ tiuch so 


tioners of behavior mod: There was no discernible effect, 


U 


at all, + of the $6 million prosram. The teachins of the 


tests themselves arrears to have been necessary to achieve 


the demonstrated effects. 
4 - ‘ 
NAEP’s .emphasis on obJective-referenced testina. = In 


lisht of this backsround, then, perhars it is much more un- 


a 


derstandable why, in the concertion of NAEP during this per- 
Ge . a 


iod of the mid-1960's, there should be so much araument in 


Favor of obJective-referenced testins (ORT) us. norm-refer- ih 


enced testing, (NRT). This shift in. type of tests was pre- 


sented to be one of the sreat: contributions of NAEP, as com 


5 


- 16 -x 


practi-— 


Ps 


a] 


net 


pared, say, ‘with,the ProJect TALENT. which had Precede* it. 
“7 2 . , : ee 


I remember larse halls at national meetings | of scientific 


. 


societies such as BERR or APA, where -Raleh Tyler and others 
apaige PpreuPulls about: the xlountanes expected, if only.the 
fonddas agencies came in For substantial amounts (and it 


was very clear BRAGS NG “important fonars Followins the 


PumP— priming of eavueaias would be j the Federal Government). 


‘ s . 


This is;,no Place to do a technical reanalysis of the dif- 


Ferences, “real or ‘imagined, between the two Kinds of test- 


: og 5 ea . ca 
ins. On one. hand there are hooks such as James Porpham‘s, - 


& 
. 


makina strons arsuments For the ,unisueness of ORT. On the 
other,’ there are the standard works on measurement and test- 
' —— ae et 

ina theory, such-as Stanley & Hopkins’ (19 ), puttina these 


arauments in careful perspective, yet trvina not to offend, 


. . 4 


at the same time, all the potential bookK~users who have be- 


4 ' , 
lieved the various arsuments for ORT. Let us. briefly con- 

é & ra eer ; sf 
‘sider some of these; _ i : 


»* “Freee Ape 


NRT ts weePul ‘only For describina the differences between 


individuals. ORT,» in this view, ean really describe “Ghat - 


Students can do," whereas mee only says whether | one student - 


is better. than another. This is simply pot. an accurate de- 


-seription of NRT, when it it is well done by competent test- 


“ing agencies. Indeed, charters in standard books on testins 


have lons showed technisues of item analysis which would 


sive the same sort of Peeiienay descriptions Por ‘the tested 
‘ 


~ 


ra 


* machine-scorable multip e-chotce item; peg 


4 


i 


’ : : , ts ’ 
_ Glass, school, prosram, or other sroup of interest, e* ap 


claimed for ORT. -On its part, ORT une no special way oF de- 


scribing what.. students can dé. It is hard to ‘imagine an 


_ os 


item constructed For ORT which would not be just as “appro- 


priate in NRT -- in facty a Jjudse could not ordinarily tell 


* oY 


which Kind of test an‘item came From, if it were a standard, 


# « 


a pnt pee ————— 


ae ‘ = * 
‘ LY = . ‘ -' ‘ 
NRT relies'on multipNe-choice items. In a-view expressed 


For ORT, one could sgmehow set poclearer picture of “what — 


. ‘ 


students. can do" if one broke away From the multiple-choice 
methods. hd in NAEP’s early days it was common .to hear 


strona calls to Professionals for new ideas, For innovative 


* methods of testina. — At .NAEP, For example, ° writins samples 


were @ollected From huse numbers of students to Find out --., 


“never mind what the,old normed tests said -- whether they 


. 


could “really write." But then what? | Then various—educa~ 


tors and specialists were:called in From across the nation 
. wo Ng ® a 


*to0 Find out what to do with all these essays. There iss. of 


,course, no standard answer. ‘When such essays are eValuated. 


by experts in writing, ‘their slobal (or holistic) 


~ e 


ments asree about 5 with each other, and contain far less 


¢ ‘ . ‘ «& 
information about specific student abilities and skills than 


’ 


“ : 
one would derive from careful analysis of a well-desisned 
a vs 


standardized test on srammar, Punctuation, spellings, and the 


° ey tal 


» like. (IF a student does-not use a suote.in an essay, -how 


© 


will a Judse decide hil competence in use of quotes?) [> 
LJ . . 


5 4 a . ¢ s 
5 - 18 - 


-_ “19 


eg 2 
aad e 


’ ar) 
.. 


deed,/ NAEP has in practrice deregded sreatly (and properly), 


on multiple-choice testing, with all its difficulties. 


8 
¢ a 


NRT is not eos Nad to Prosram effects. As we have ~ , 4 ¢ 
noted in the historical eevee this charae is _lersely true. 9 fe eee 


e —_ 


Most standardizéd tests are desisned to measure somethings 


. i 1 ‘ . * : : i 
that is ceeeacnapakeiie Ai the eratey individual. © The 


‘test is resarded as a Kind of random sample of items From a 
virtually infinite Population of items: measurins the hue 
ability. A short-term trainina Prosram, thernFore, 
_hot be expected to aitexs except marginally, this“ability 


(whether We were cénsidering an individu Ya Sroup of in- 
r : * Fe " ry ¢ 
dividuals). Therefore it As “try 

. , oe aciwilng! 

ic trainina session (sayr on the use of quotes in Enslish = 


hatr~for a hishly specif-— ; 


writing), we can desian test items to measure Student,acaui- 
: Me 
° - » 


.sition of the objectives. This is commonly done by able 
teachers in the classroom. And most standardized tests in 
wide use will have at least a ‘short sample of items” on 


auotes. Can ORT aa more than this? It isshard to’ see, hows 


and NAEP! S$ own record does not reduce the skepticism. 


« 


‘What we’ seer then, ‘i& that the emphasis on obJective-re- 
Ferenced testing sreu aireahae From the social drives of the 


period which sane ‘neh ¥ NAEP. Unfortunately, ORT has 


some paihet larse diicadyanianias which had better be noted . / 
“here as well: | | _ . | gre ate 
' Le . ; x 
a 19, = » @ e, 


In NRT, 


the, theary 


ity\estins is Fully developed and very useful. We | ‘ 
“easily \¢ lculate means r standard deviations, “and corre-. 
} a 


\ } eo as 3 
ations amons abilities. We can use technivwues to estimate 
the ranses oe the true scores, _ under Vira the. observed ‘ 


scores. All of aatpa- ~curve ‘theory comes into play. We can 
‘ is pres, rs - iets = 
describe contrasts etude STOUPS) and proarams } in well-un- 
ee ee nti eae aS t 


‘derstood ways (For exanirle, in terns of means ary more Pou- | 5-9 


. 
a . 
. 


erfully, BF median gueriar). In contrast with such uses of 


_ -t— Se eee 


* NRT, ORT is reouliahly henweruan oy\its a to Justify its | : 
we 
test Philosophy. NAEP ‘ eT “therefore, often consist. ‘ 
r , \ : we : aoe n 4 nan * 
of item-by-item response. Freauencies For Srours,* and the 


oh, 
larser, more efFicient comparisons are lost sisht of. 


Y : “Sp oO , 
ORT is not statistically tractable. This is .no lisht 7 : 
charse, For the truth of “national prosress," once one steps y, Ue A 


away From the straitjacket of .obJective testins, must rely 
heavily ont Advanced statistical methods. §' The trackina of 


chanse across time (For example in study of the test-score 


F880 aos 


“decline? must depend on having the best statistical tools at 


ci 


i . 4 
our disposals and there is nothina in ORT to compare with 


the brilliant normal-curve theory we have inherited From De- 
9 , ’ . ‘ a 


+) r . 
Moivre, Galton, Spearman, and our other prosenitors. 5 tw 


* « 
« 


* . ; f ¢ , : i . 
ORT is awkward ‘in causal study, _After all, qthe Purpose 


of NAEP is to rrovige infarmation €or national decisiak-mak- 


“ieee And decisions depend directly on LWPorwat ion about the 


ww * 


* 


janet of’ student achievement. ; “apart From experimentation: 
(which is virtually impossible to sonduad erPielentiy in the 
schools epdavns our best methods For exploring causation. 
currently come From path analysis -- the collection oF weihe 


ods for estimatins the influences of variables on each other 


(cf. Kenny, 19793" Heise, 19753 and many others) All path 
anlyMitic ne Tygds—tesins with. correlations or “covariances, 


; ~ 
which depend on, havins scores which Permit the’ expression of 


. 
os 


egen relationshies. I ‘know of hardly any causal studies, 


_ ee + 


using ‘such powerful hhethods, to have come out of .NABP (in 


. ee 
La cy 


strikings contrast, For example, with the work done with Pro- 
ae me 


Ject “TALENT > or. with the more recent Nationa¥Y Lonasitudinat—> 


te 


“Study, or ‘with High School: and Beyond). This is a ne exe © 
ious-limitation of the obJective-referenced Framework “--a . 


oe Ra a be Us : s 


Limitations moreover, strikins at the heart of ‘NAEP ’s Faund~ : 


ing Purpose. : ‘ 
. 1 @ 4 mS sae 


a ee ee NAEP , } ‘ i, +e 
w ie) os Ay 


Just as NAEP’s measurement theory sarew out of the 1960/5 


“and the behaviorist movement, $0. did ‘NAEP “s tanpling strate~ 


. 


gies reflecét the same spirit. If achievements are a collec 


\ 


tion of specific: skills, “to be trained and tested, then .it 


’ boa 


_is less AMPETEAGE to Know much’ about the’ endive 1 differ- 


ences oF the learners. What is-more important is to measure 


. 
e 


those seecific skills. The apineieles of. NAEP included a 
severe: limitation (one hour) 6n the time .reauired of any one. 
~~ : 


- - 


v 


ee \ 9 “i 
student. Rathers different items and measures were applied 


to different sroups : of students ~~ bith only» a skeleton of' - 


backsround information on each participant. In the same) 
spirit, there was no effort, to look at the developmental va- 


° 


riables best stifdied hy repeated investisations of the same ‘ 


- . i . 
« htt ib . ie ; 
fp ae students across time. In its sampl'iina strugture, once 
a Pe ae >, ; 
aT: asain, NAEP has resembled.the behavior modification srour of 
ae psycholosists relatively more than those in measurement or — < 
° . —— . : ‘ \ a aa 
” a child development. sg “9 


e ow 7% : x \ ; . 
- From the standpoint, of research value, the loss of infor- 


\ 


; %, . % mation From the NAEP sampling strateay is incalculable. The 

% information From very short vectors ‘OF date (with Few items 

.- - i . on each student) is slisht -- ena 43 is no wonder that there 

; ’ - has — weiny huh Sertaus research use made.of NAEP ’s 
data, c pared with smaller and less expensive data sets of » 


J , eo 


2 ; 
3 ¢ ‘ a more measurement-oriented design. 


ra 
. 


, -  ,, What is often not fealized is that information iis roushly | , 


: ; , “proportional not simely to the number of information items 
; on each subJect, ;, but proportional to the saiiave of the num- 
"5 , ber oF “items, * This happens because available er ce 
, ~ may be thousht af as the number correlation coefficients ; 
eossible From a dataset. The actual Formula is"Qhat! ‘oe 
a eer | -. “eR E2 bs re Eauation (1) 
4 . ~ o7 


« “oe 


e P " » on 


where n is the number of items on each student, ‘and where c 
“. . ¢ a 


> ow 
is the nutiper of bivariate. correlations obtainable from the 


“matrix. ‘This is: not to mention’ the much larser number of 
: bee 


three-variable and n-variable . complex relationships expres 


. 


gible From longer data vectors. So it is clear that all 


those interested in research -- notably university research- 


e 


‘ers and scientific societies -- are losers by the samplings 


‘ 
Y 


C4, ‘ 
stratesy adopted by NAEP, 


ss ns * af 


is that virtually all potential 


t e 


What is not clear to many 
users of, NAEP’s output are similarly deprived. As we have 


noted, most of our information needs are concerned with 


causes, hot simply with observed data.. And decision makers, 


wish .to.Knouw the probable effects resultins From-:their 
| | Se 
choices. ‘Simple-minded one variable and two-variable data 


| 
reports are not apt to meet anvone’s needs, for they are of- 


‘ een So ee ee 
ten as misleadina,: From the standpoint of decision makina, 


‘as they are informative. 


¢ 


The principal message oF this portion, then, ,is that 


NAEP‘’s samplina desisn, like its measurement theory, needs 


to be rethousht From the sround up, and Freed of some ‘of the 


° . 


edoctrines implicit in its foundina. 


Se Ls ; 


The Problem of Access to NAEP. Data 


Po 


Obviously, data are not valuable which are not analyzed. 


What *is less pbvious is that, . siven a truly rich data set, 


4 


vo 
> 


jae. 


as : * : . se ig, . j 
2° . it is impossible to predict all of the useful avalyses: Ae A Ap 


» 


* the cantrary, the serendipitous discoveries From data can, 


' . 
the orisinal intention. 


at “timesr i ‘fruit surpassing. 
. iu "S 


» % 


And, depending on who is doina the analysis and on what as- 


sumptions are made, different. rae ech i can result in 


strikinsly different conalusions about the results, and 


oe 
¢ 


hence about the arrpropriate decisions to recommend. (Eyaok : 

by usins different controls Pase & Keith, 1981, produced 

‘very different comparisons of U.S. pubs and private hish > a 
- i So 2! ‘ 


schools from those drawn by Coléman et al., 1981.): 
} 


. i 
' = . eo . 
. 


. 
nO a 


5 In the old, experimental models of the psycholosical 


& 


‘ e er Past, the data were seldom available to outside researchers. 


3 @ And even the monumental Project TALENT did not Produce ‘data 
. * : ‘ vag : " i) 
7 tapes pasily available to researchers of different Purposes 


a 


or views. For the educational ‘research community, the First 


s 


. breakthrough of access has been with the National Lansitudi- 


es 


nal Sthdy- (NLS) of the Hish School Senior Class of 1972. 
l The NLS has operated exactly From the viewPrpoint expressed 


at here: ‘that sood data sets are inexhaustible and may be 
mined richly, and,indefinitely, by a host of researchers For 


whatever Purpose. Thus, the NLS (sponsored and mofiitored by - 
\ 7 . 


the NCES) hag emphasized the availability of data much more 


than its analysis. With encouragement to researchers, with Y 4% 


a 


low costs, and, with widespread information about the NLS, 


‘tonether with the base year (1972), and four Follow-up years 


(the latest beina 1989), “For a total data vector of some 


4 


‘ 
. 


’ 


te 


argan cane riches from it,* in basic or applied educational 


~R & D agencies. It is a Knowledse contribution which cannot. 


2,000 neasunes, F ene NLS has attracted sreat interest and 
° , 4 * 


‘ * ; J * 4 
use. - + . H . 
- : . 4 


, ae 


* 


. . . a 4 P ‘ 


The NLS‘s direct successor has been the stil] richer Hish 


School .and Beyond (HSB), which studied 58,000 hish schoal 


~ 


senYors*and sophomores in 1980, and has already put in. mo- 


. = . 
2 ° 


‘ ‘ ° 
tion-its First Follow-up (1982). I have arsued elsewhere 
XN 


that HSB may be the sinsle best event that ever happened fo 


- . ° . 


educational. and psychological research (Page, 1981). Amons ee 
‘ a : : A 


other innovations, it is a ‘“areat leveler: Anyone can set | v1 
- . a as “« 


- 


ta ” 


% : : 
the data for low cost. A brilliant sraduate student may  . 


. . 4 . 

Knowledser than a distinsuished professor or hedd of a larse 
. ‘, # . . of : 

institute. It permits the orisinal analysts, tosether with — -° 


their critics, to compete on equal terrain, thejr arms con- 
‘ 


sistina in their ability to: think, analyzes and report. 


Journal editors are clearly hunsry for the Kind, of seneral- 


° 


izability which such data sets make possible. . 


a ‘ . 


And decision makers, from state and local ‘and Federal 


c e a . ’ ‘ . 
‘agencies, can explore the data with their particular sues- 
1 7 - x Fi 


tions in mind. These will very often be different From the. 


4 


. 


iw 


~~ . 


routine information siven out by any one seneralaasency. OF 


course, it remains a problem to establish skillful data ana- =, 


. ’ s 
lysts $0 hele decision makers,, but such analysis may be . , 
% . E ’ 


drawn From skilled researchers in universities, or in other 


- . 


al ~ 


« 


be overemphasized, made by the openings of such rich data to 


6 


: Universal scrutiny. It is also the mark'of a remarkably 


conf itent. and open Form of sovernment that a nation is will- 


i : ing to do $0. 
‘ 


a ’ * 


In light of these developments with the NLS and with HSB, 


may the NAEP archives be made equally accessible? One of 


NAEP’s most valuable recent chanses is in the plan to pro- 


e 


duce widely useful Public data.tarpes. Already some of these 


are in place, and one may expect many more in the Future, 


Siven the strons intention, within the NIE, to achieve this 


2 


. soal. These steps are to be sredtly encourased} many of 
™* , these data sets, will be unique For some maJor lines of in- 


7 vestigation (for example, the promised tapes Friom the writ- 


- ina samples, in machine-readable Form). 


. 


Nonetheless, one liability should be pointed out affect- 


< \ ins NAEP Far more than these other data sets. — As we have 


ry 


é ; os 
noted,. NAEP’s use of many different student samples means 


that researchina complex relations Will be much more difFi- 


(1), noted above, will be a much smaller Fisure. In future 


aaee " lversions of NAEP, it is hoped, a maximum amount of informa-— 


tion will be collected from smaller numbers of students, 
: F PY e ii $ 
ee Then the sreater access to NAEP data will be of much larser 


. 


benefit for researchers, for SEAs and LEAs, “and For all 


. 


. those decision makers, Public and privater with a stake in 


ig 


American education. 


‘ : is se es : a 
cult, because the coefFiciént of information C in Eauation 


. 


—- 


, RECOMMENDATIONS FOR NAEP’S FUTURE 


e - * 


. 


From the present, analysis, and From other considerations 
‘ 


t 


as well, there are a number of recommendations which may en- 


“hance NAEP’s Future utility, For°all of the tarsets oF its 


concern: federal, state, and local policy makers, profes-_ 


sional associations, the research community, teachers, stu- 
as : ; 


dents, and the public... These recommendations will be here 


made under the headinss of desian strateay, sampling, mea- 
, F € A 
surement, reporting, dissemination, and administration. 
a 


* ‘ rm bs 
Desisn Stratesy : ‘ 


< 


1. Look for all explanatory causes of important student 


behaviors. Include influences of the family (both 

environmental and biolosical), of the church, neish- 

. borhood, and other major influences outside the 
& F 


school. 


2. Continue .to stress curriculum in its relation to 


learnings. Include not only courses taken, but home- 


¢ 


Ld 
ay 


work reauired and homework actually done. 


3 - 
3. Save desisn money by usina the lower-cost multiple- 


___thoiee-items—uhere appropriate, including those From 


‘ : 7 t 
instruments already available and on-the market. 


‘a 


. 


: 4. ‘Also save costs throush plannjna the use of —Pewer 


4 s , Student subJects.with more ‘contact time per student. 
PB ge a , (These points are repeated below.) . 
ne Saupline Issues tage: ' 
: : ‘ 1. Abandon wost of the separate student samples. Move 
8 i : renin 


¢ 


rs | - » toward more examination of Fewer students to create 


“ssthe’ most’ informative data matrix possible. : ; 
on 


‘emphasis on the school years, however, and leave to 


o e . ' 
ts other Programs any adult -follow-ur, | = ” 


3. Continue matrix samplins of achievement, where appro- 


‘ g ® 2 x 
i. ° priate and necessary.fo economy and reasonable data 
{ burden. But use the economical standardized tests 


ite : wherever Justified. a 


, "4. In “the sampling of ‘students, include a sood number of 
ape a ; a 
7 students. of hish ability and achievement (as well as 


a 9 


other tarset srours) since these aré important to na- 


i. 


tional Productivity and .. other national interests. 


5. Include a larse’ sample of siblings and twins of the 
saa : : : : ; 
- student subjects, for better explanation of home and 


a - 


of school fects. 
i 7 , 
- + * 4 ‘ ~ 
Measurement Issues 


. e 


a 


» = . 


¢ ~- 


a 


te f ized prosrams~. and From other researches, would be 


‘.* < 
= 


e Sh pe 
. 
e * et ‘ = 


, . ’ 


- 28 - 29 : 6 


i a “2 w . ® 4. . 


2. Examine- students loWsitudinally, showing educatinal — 


and other srowth across the school vears. . Keep the 


g 1. .As noted, available instruments, both from standard- — 


economical replacement for many. items created by 


se 


NAEP, and should be ued. Continue to monitor educa- 


~* . 


. 


tor and lay opinion about ‘the objectives of educa- ' 


tion. Use these Findinas to help structure inayviry 


into skills and curriculum, and to validate the tests. 


° . 
” 


chosen for use. : . . = ‘ 
« eee f . . : t 
2. At the same timer. recosnize the essential samplins oo 
1 ’ 4 i P ° o . 


nature of all achievement tess, the heavy loading , 


\ 
of’ seneral ability in such’measures, and their normal 


\ . 


distribution. For each Student, report scores which 


f RS og rv ‘ . 
” 4 may be used: in ipa ways for pxplanation eye 
1 ‘ ‘ 
p and prediction. : 
- : t 7 ah Rly ‘ = 
3. Measure a ranse of different: sub-tests of ability, 
S F 2 Fat eee y ee v a8 : ° 
includina ‘verbal and nonverbal items of intelligence. 
Use auestionnaire items For extensive information : 
_~ “about social, familial, ethnic, and physical and | 
health characteristics of the students. Asain, these é ‘ 
4 ; os ee & ’ 
* may include -standardized or other avVailable-instru- 
. ments wh@re appropriate. a a 
: _ ; ' 
. : 4. For twins of the same sex, determine ‘their Kinship 
‘(whether identical or, Fraternal). Such information 
‘ os can help -in causal explanations of school achieve- 
ments, and of other behavior. * 
Jos, > ; ai iat ee 
Reportins Results - , : oe 
‘i : ; ee ® 
. ‘1. Continue to mail out destrirtive material about the 


studies and their most general results. 


. 


1. 


e 4 


only samples 


Dissemination of Data 


" 


a2 


In addition, howeverr “solicit some advanced causal 
analysis) of key issues, to Se ai laa and. 
use of researchers and policy-makers. Welcome diffe- 


rent analyses and. debate in the Public Forum. 


| 


For the SEAs and LEAs¢ report information, on current’ 


changes discovered by the latest surveys, in student 
6. ; F ? : : ; - ‘ 
attitudes and achievements, and’ -in school orsaniza-. 


tion and curriculum. 


. , 


In reportins, Keep’ in mind that education seeks sen- 


eral liftina of student abilities, that items are 


From these larser traits *and achieve- 
ments. a + ee , , 

* , . 
In srourp comparisons, use more informative indices 


° 


tHan “percentase correct." ‘For continuous traits, 


usa such comparisons as median overlap, or simple 
‘ t 


‘phrases carrvina.the same meanins. 
2 ao. 


In seneral, abandon the fansuase of passins or fail- . 


ina. There is no such deFinable line in the wordd of 
normally distributed, latent.abilities. 

Where possiblé and reasonably consensual. show the 
causes of differences noted. In practice, this may - 


e ; . 
mean approximated by.showins differences after cont- 


rolling for backsround variables. : 


f 


Follow throush with present Plans for clean, well-do- 


’ 


cumented data tapes available,at very low cost to any 


reauestina users. 


* 


* 
° 
. * ” e. we 
é . b ‘ 
r 


2. Orsanize these tapes around the. larse body of exten-_ 


ro ° ‘ . + 


stvely researched students described above. .That is; 

in the recommended Future structures these dili'not . 

TSE aoe | bie “ususily ‘orsanized around serardte curricular is- 
» . ‘ 


’ ¥y 


3. For those different data sesments already collected, . 


organize material$’ with extensive description which 
5 may be widely announced to prospective user SrouPs. 


é Y 
. * . 


é 


‘Administration of NAEP . oe 


1. Continue advisory srourps From informed professional 


and lay constituencies, but also experienced re-:. ° 


searchers and measurement personnel of the hishest | 
. . 1 7 


- quality. - This should make for a- sounder scientific 
“opdentation in the NAEP oreration. ° ; 
. :. a ‘ 
2. In the scientific advisory role, include more experts. 
"experienced in “ane dedicated to causal analysis and. ; 
\ 


a \ 1 


to the norm-referenced tradition. 


3. For economy. ‘and efficiency, ut the NAEP contract 
o periodically out to “rebid. As recent experience — 
‘ shows elsewhere, competitive biddins may shave more 


than a million dollars a year’ from the annual cost of 


: a es ? : 
Prostams of this masnitude, .-- ~ | ‘, + A. Res 


a 


¢ 


. 
—_ . 


3 : é 
In summarys As thinkers have'recosnized since Plato, 


there is ‘no more important task of sovernment than educar 


tion. Its process, and prosress, must ba‘Freauently moni- 
F a * . i: . 
tored by society at larse. We now possess imeprovins methods 


for makina such: monitorina’ effective, and For interpretina 


. 
. 


> 4 ot : 
the results in meaninsful and useful ways. When ‘orsanized 


4 


well, information systems may provide both simple-and com- 


plex analysis ~ to deeren our seneral -understandins oF the 


educational process. And throush the use of simpler. lan- 
suase, these.more sophisticated _interpretations can be ex- 


Pressed to the targets of all our concern. ‘the SEAs, LEAs» 


’ te . - ed o> 
schools, educators, parents and children and citizens -- to 


211 wha sh to improve the quality of human life and: human 


learnins. 


. F 
. e . 
oun 
acetate eh th rth be Oe ee ree Ga a s 
; ~om 
3 
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¢ 
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